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BATBook:  An  Online  Book 
and  Problem  Solving  Environment 
for  the  Study  of  Skill  Acquisition 

Jeremiah  M.  Faries 
Brian  J.  Reiser 
Princeton  University 


Abstract 

BATBook  is  an  online  book  and  problem  solving  environment  that 
facilitates  students’  use  of  examples  in  a  text  book  and  use  of  their  own 
solutions  to  previous  problems.  BATBook  contains  facilities  for  reading 
and  searching  text  and  examples  within  the  text,  for  working  on  prob¬ 
lems  and  storing  solutions,  and  for  searching  through  past  solutions.  All 
interaction  with  the  system  is  recorded,  and  thus  BATBook  serves  as  an 
experimental  tool  for  studies  of  the  use  of  examples  in  learning.  BAT¬ 
Book  has  been  used  for  studies  of  skill  acquisition  in  the  programming 
domain,  and  can  be  used  in  similar  fashion  for  other  domains  such  as 
statistics,  mathematics,  or  physics. 


1  Introduction:  The  role  of  examples  in  learn¬ 
ing 

An  important  focus  of  research  on  skill  acquisition  is  the  role  of  examples  in  learn¬ 
ing.  Students  rely  heavily  on  examples  in  instructional  text  to  learn  procedures  for 
solving  problems  in  new  domains  (e.g.,  Chi,  Bassok,  Lewis,  Reimann,  &  Glaser, 
1989;  LeFevre  &  Dixon,  1986;  SweUer  &  Cooper,  1985;  VanLehn,  1986;  Zhu  &  Si¬ 
mon,  1987).  Research  on  the  early  stages  of  learning  has  investigated  how  students 
use  text  examples  and  their  own  memories  for  previous  solutions  to  adapt  for  new 
problems  (e.g.,  Anderson,  1987a;  Escott  &  McCalla,  1988;  Faries  &  Reiser,  1988; 
PiroUi  &  Anderson,  1985;  Reed,  Dempster,  &  Ettinger,  1985;  Ross,  1984;  1987; 
1989).  Research  on  analogical  reasoning  has  considered  the  mechanisms  that  ac¬ 
cess  potential  analogues  from  memory  and  evaluate  them  for  their  application  to 
a  current  situation  or  problem  (e.g..  Centner,  1988;  1989;  Holyoak  &  Thagard,  in 
press;  Thagard,  Holyoak,  Nelson,  &  Gochfeld,  1989).  Recent  work  on  case-based 
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rccisoning  has  explored  models  of  expertise  consisting  of  an  organized  memory  for 
individual  cases,  rather  than  merely  consisting  of  generalizations  (Hammond,  1989; 
Kolodner,  1983;  Riesbeck  &;  Schank,  1989).  Research  has  also  focused  on  the  types 
of  features  that  govern  memory  search  for  previous  examples  (e.g.,  Faries  &  Reiser, 
1988;  Centner,  1988;  Ross,  1987;  1988;  Seifert,  1988;  Seifert  &  Hammond,  1989). 

Several  difficult  issues  face  research  on  the  use  of  examples  in  learning.  First,  if 
the  goal  is  to  characterize  how  examples  are  used  to  learn  rules,  plans,  or  procedures, 
then  it  is  important  to  study  the  use  of  examples  during  the  problem  solving  process. 
Therefore  it  is  necessary  to  have  subjects  learn  to  solve  problems  in  a  new  domain, 
and  to  examine  the  learning  in  a  period  that  is  long  enough  to  include  a  sequence  of 
problems  in  which  earlier  solutions  are  relevant  for  later  problems.  It  may  be  difficult 
to  generalize  from  experiments  giving  subjects  only  material  to  read  and  remember 
to  what  happens  when  students  are  learning  to  solve  new  types  of  problems.  Second, 
in  many  experiments  it  may  be  difficult  to  determine  when  subjects  are  using  an 
example  in  a  text.  Some  method  is  required  for  monitoring  which  pages  of  a  text  and 
which  portions  of  the  pages  subjects  are  currently  \ising.  Third,  even  videotaping 
students’  use  of  a  text  does  not  make  available  the  information  the  subjects  used 
to  access  previous  text.  Fourth,  it  may  be  difficult  to  distinguish  a  case  in  which  a 
subject  has  retrieved  a  memory  of  a  past  solution  and  applied  it  to  a  new  problem 
from  a  case  in  which  the  subject  haw  learned  a  general  procedure  and  then  applied 
that  to  the  target  problem.  In  general,  an  optimal  methodology  for  the  study  of 
examples  would  make  it  possible  to  investigate  in  an  extended  problem  solving 
context  the  information  subjects  use  to  access  examples  and  past  work  as  well  as  to 
monitor  what  information  from  the  text  or  past  work  subjects  use  in  a  new  problem 

One  answer  to  several  of  these  problems  relies  on  protocol  analysis,  in  which 
subjects  are  asked  to  talk  aloud  while  trying  to  solve  a  problem  or  understand  an 
example  in  a  text.  Protocol  analyses  are  one  approach  to  gathering  more  fine-grained 
data  for  use  in  building  and  evaluating  learning  models  (Newell  &;  Simon,  1972; 
Ericsson  &  Simon;  1984).  For  example,  Anderson  and  his  colleagues  have  studied 
students  learning  to  write  computer  programs  or  to  solve  geometry  proofs  and  have 
examined  30-40  hours  of  learning  time  per  subject  (Anderson,  1982;  Anderson, 
Farrell,  ic  Sauers,  1984),  and  Chi  and  her  colleagues  have  made  a  similar  study  of 
students’  elaboration  of  examples  of  worked  solutions  while  reading  instructional 
text  (Chi  et  al.,  1989).  However,  it  is  often  desirable  to  both  restrict  the  range  of 
data  gathered  and  to  find  a  somewhat  l^s  intrusive  method  for  collecting  the  data. 

In  this  paper,  we  describe  the  Behavioral  Analogy  Tracing  Environment  (BAT- 
Book),  a  computer-based  environment  in  which  students  can  learn  to  solve  problems 
in  a  new  domain.  BATBook  is  an  online  book  and  problem  solving  environment  that 
makes  the  search  behavior  of  students  explicit  and  provides  a  general  method  to  ex- 
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amine  how  students  use  examples  in  text  and  memories  for  their  previous  solutions 
during  learning. 


2  BATBook;  Facilitating  and  Monitoring  the 
Use  of  Examples 

BATBook  is  an  electronic  book  and  problem  solving  environment  designed  to  facil¬ 
itate  and  monitor  students’  use  of  text,  examples,  and  their  owu  previous  solutions 
to  problems.  Thus,  BATBook  is  designed  to  be  both  a  pedagogical  tool  and  an 
experimental  tool  for  the  stud>  of  skill  acquisition. 

2.1  Functionality  of  BATBook 

BA'T'Book  can  be  used  in  skill  acquisition  experiments  to  present  instructional  ma¬ 
terial  and  monitor  students’  problem  solving  behavior.  The  bztsic  paradigm  is  that 
students  read  a  textbook  on  the  computer  screen,  interspersed  with  example  solu¬ 
tions  to  problems  and  with  problems  for  the  student  to  solve.  Students  progress 
forward  and  backward  through  the  text  by  pages  and  can  search  the  expository  text 
and  worked  solutions  for  any  particular  content  they  can  specify.  While  working 
on  exercises,  students  are  free  to  search  the  instructional  material  in  the  text,  the 
examples  contained  in  the  text,  and  their  past  work.  Students  store  their  completed 
solutions  and  can  request  to  see  a  previous  solution  at  any  time.  BATBook  records 
all  interactions,  including  the  time  spent  reading  each  page,  all  the  successful  and 
failed  searches,  all  problem  solving  work,  and  searches  of  examples  and  previous 
solutions. 

We  have  used  BATBook  to  study  the  use  of  examples  by  students  learning  to 
write  computer  programs  (Faries,  1988;  Faries  &  Reiser,  1988;  Faries  &  Reiser,  in 
preparation).  This  version  of  BATBook  contains  the  text  of  the  first  three  chap¬ 
ters  of  Essential  LISP,  a  textbook  on  the  LISP  programming  language  (Anderson, 
Corbett,  &  Reiser,  1987),  and  a  problem  solving  environment  consisting  of  a  LISP 
interpreter,  a  simple  editor,  and  facilities  for  checking  the  students’  solutions  for 
correctness  and  providing  simple  feedback.  In  addition  to  searching  the  text,  the 
examples,  and  previous  completed  solutions,  students  can  also  search  their  inter¬ 
actions  with  the  LISP  interpreter.  In  this  way,  the  BATBook  environment  makes 
explicit  a  large  portion  of  the  student’s  problem  solving  behavior,  providing  a  rich 
record  for  analyzing  when  and  how  students  access  the  written  instruction  and  their 
previous  work.  Students  learned  to  use  the  BATBook  system  with  little  trouble 
following  a  short  demonstration  of  its  capabilities,  and  worked  with  it  between  10 
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and  15  hours  to  learn  the  three  chapters  worth  of  material,  making  effective  use 
of  the  searching  capabilities  the  system  provides  to  help  use  the  text  material  and 
their  own  solutions  when  working  on  new  problems.  The  following  sections  describe 
the  various  components  of  the  BATBook  environment. 

2.2  Implementation 

The  BATBook  environment  runs  on  Sun  workstations  in  the  SunView  window  sys¬ 
tem.  BATBook  is  written  in  C,  Franz  LISP,  GNU  Emacs  LISP  (Stallman,  1987), 
and  the  Lex  lexical  analyzer  (Lesk,  1975).  The  majority  of  the  interface  for  display¬ 
ing  and  searching  text  is  written  in  C,  using  Lex  string  handling  functions.  The 
problem  solving  environment  for  LISP  is  written  in  Franz  LISP,  including  a  modi¬ 
fied  LISP  interpreter  that  records  the  student’s  interactions  and  checks  completed 
solutions.  The  current  version  of  BATBook  contains  a  total  of  5800  hnes  of  C  code, 
3200  hnes  of  Franz  Lisp  and  GNU  Emacs  Lisp  code,  and  90  lines  of  Lex  code. 

We  have  used  several  different  versions  of  BATBook  in  our  experiments.  The 
exact  configuration  of  windows  and  the  searching  options  available  to  the  students 
differ  from  one  experiment  to  the  next.  In  the  following  sections  we  characterize  the 
various  options  available  in  the  systems  used  to  date,  which  are  designed  to  include 
a  LISP  programming  environment,  but  we  also  describe  how  BATBook  can  be  used 
to  teach  topics  other  than  programming. 

2.3  BATBook  Windows 

The  standard  BATBook  screen  contains  a  Text  Window,  an  Exercises  Window,  a 
LISP  Interpreter  Window,  and  a  Problem  Submission  Window  that  appears  when 
the  student  stores  a  completed  solution  (see  Figure  1).  The  textbook  is  read  and 
searched  in  the  Text  Window.  The  book  and  chapter  title  are  displayed  as  the  title 
of  this  window.  The  Text  Window  is  replaced  by  a  Previous  Problems  Window  when 
students  want  to  look  at  text  examples  or  previous  solutions  (see  Section  5).  The 
Exercises  Window  presents  problems  for  the  student  to  solve.  In  the  LISP  domain, 
students  construct  their  solutions  and  test  them  in  the  LISP  Interpreter  Window. 
For  other  domains,  this  window  would  be  replaced  by  another  type  of  interface  for 
constructing  solutions,  such  as  an  equation  editing  interface  or  a  simple  text  editor 
for  constructing  verbal  answers  to  problems. 
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IS&IHIIAL  LIST:  Oiaptcr  2:  IIST  F« 


Ucal  ani  filobal  Varlablaa 

Parmt«rs  tri  typically  tM  hardatt  tnmg  to  grasp  in 
daflnlng  na«  functions.  Psaeabar  that  paraaatcrs  art  actually 
varlablas.  Houavar,  aa  do  not  assign  thaa  values  sith  the  function 
aatg.  Instaad^  each  paraaatar  is  ttsignad  a  value  autosatleally 
Hhan  the  function  is  called.  Specifically,  each  variabis  is 
assigned  the  value  of  one  of  the  arguaants  In  the  function  call. 
Since  paraaetars  era  verleOlas,  paraeaterc  are  eleeyt  etoas, 
regardless  of  uhathar  the  value  of  tha  arguaent  that  sill  be 
assigned  to  than  is  a  list  or  a  nueber  or  a  non-nuaaric  atoa. 

There  la  another  laportant  distinction  betaten  peraaeters  and 
the  variables  «e  discussed  in  Chapter  l.  Paraeetere  are  local 
vanabias,  shiia  tna  wanabiet  in  Chapter  i  ere  global  vanabias. 
Paraaetars  are  called  local  variables,  because  they  only  have 
valutt  Hithin  the  context  of  the  function.  Considor  tho  follouing 
oxaoplo: 


:>  (dafua  doublt  (nta) 
(•  ttm  2)) 

doublo 

s>  (doulila  7> 

14 

s> 

Unbotftd  variable:  nua 


I4i#n  iaabla  aes  called  in  thie  exaaple,  the  paraattar  aw  vae 

aaeignad  the  value  7«  but,  after  the  function  cell  hat  oeon 
evaluated,  if  you  try  to  get  the  value  of  ims.  you  got  an  error 
sassage,  because  that  value  oas  only  assigned  to  the  persoetsr 
during  the  execution  of  doabla.  after  (double  7)  ues  evaluated 
and  returned  14,  the  peraaeter  nia  looses  the  value  It  see  given 
~  It  sgain  becoses  unbound.  The  vslus  a  pa'-aastar  scqulrat  is 
therefore  locsl  to  tha  function  using  tha  paraeetar.  Tha  va^'lablts 
In  Chapter  1  are  celled  global,  bectuss  they  ere  not  oat  just 
«r1tnin  tha  context  of  •  specific  function  ue  dofinad.  Ptcall 
that  unen  ue  set  a  variseia  in  Chaptar  i,  it  retained  us  value 
until  that  valut  Is  changed  eith  another  call  to  aat^. 


\\\\\\\\\\\\UU\\\\\  LISP  Intarpratar  Window  ///////////////////// 


rPravlous  Prpblensi  I  LISP  hlotor 


Click  button  to  cliang# 


Figure  1.  Reading  text  in  the  BATBook  electronic  book  and  problem  solving  environment. 
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3  Reading  the  Text  in  the  Online  Book 

3.1  Reading  and  Paging 

The  Text  Window  is  positioned  on  the  left  half  of  the  screen,  and  displays  the 
equivalent  of  approximately  one  page  of  text  from  a  printed  book.  The  text  is 
displayed  in  the  window  in  a  12  point  font,  and  may  contain  boldface  characters  for 
marking  key  words,  for  displaying  chapter  headings,  and  for  emphasis. 

Students  can  read  and  page  through  the  text  using  simple  mouse  commands. 
When  finished  reading  a  page,  the  student  can  select  the  page  forward  button, 
labeled  “+,”  whereupon  the  next  page  of  text  is  displayed.  Students  can  also  page 
backward  using  the  button  labeled  and  can  return  to  the  first  page  of  the 
chapter  by  selecting  the  “Page  1”  button.  The  design  of  the  Text  Window  was 
loosely  based  on  the  Superbook  electronic  book  environment  (Remde,  Gomez,  & 
Landauer,  1987). 

The  input  text  consists  of  a  simple  ASCII  file  and  is  set  up  like  the  input  text  for 
formatters  such  as  Troff.  BATBook  will  fill  the  text,  insert  section  headings,  insert 
paragraph  and  page  breaks,  switch  between  the  Roman  and  bold  font,  and  perform 
other  simple  formatting  using  Troff-like  commands. 

Each  section  ends  typically  with  a  series  of  examples  for  the  student  to  read  or 
exercises  to  solve.  In  most  of  our  experiments,  students  are  not  permitted  to  page 
forward  to  read  the  next  section  of  text  until  they  have  completed  the  problem  set. 

3.2  Searching  the  Text 

Students  frequently  need  to  refer  to  previous  sections  of  an  instructional  text  while 
reading  or  solving  problems.  With  BATBook,  students  can  search  through  the 
current  chapter  of  the  text  using  a  search  capabiliiy  similar  to  that  provided  in 
most  word  processing  programs.  The  student  types  a  target  of  one  or  more  words 
after  the  “Target”  prompt  and  then  selects  the  “find  first”  or  “find  next”  button. 
The  “find  first”  button  searches  the  text  for  the  first  occurrence  of  the  search  string. 
The  “find  next”  button  searches  the  text  beginning  on  the  current  page.  If  the 
cursor  is  currently  residing  on  a  match,  then  it  finds  the  next  occurrence  of  the 
string,  otherwise  it  simply  searches  for  the  string  from  the  top  of  the  current  page. 
If  the  search  succeeds,  the  page  containing  the  target  string  is  displayed  in  the  Text 
Window,  and  the  string  is  displayed  in  reverse  video.  An  example  of  a  student’s 
search  for  a  string  in  the  text  is  shown  in  Figure  2. 

A  string  of  any  length  may  be  typed  as  the  search  key.  The  string  may  include 
spaces.  Thus  students  may  search  for  phrases  such  bls  “last  element  of  a  list.”  The 
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Algorl that 

Nov  you  havi  Itarntd  to  dtfino  your  o»n  luicttont.  Tht 
rtMindtr  of  ttiU  cNtpItr  introQucti  MM  ntu  LXSP  functlOM  ond 
gl«M  you  prvctico  putting  togothtr  coabinttiont  of  function*  in 
oratr  to  Kcoopnih  Mr*  cotp^KatM  tost*.  *$  ms  Mntionao  in 
Choptf*  If  •  •uottontial  part  of  Itarning  to  progra*  involva* 
Itarnlng  not  Just  hov  functions  Mfk,  but  hot  to  u*t  thta  to 
coaputt  OMirod  rttuitt.  Tht  proctduro  by  «nich  a  function 
coaputas  a  rtauit  it  caUad  tna  aigoritha  for  trio  function.  Pot 
axaaplt,  our  algorltna  for  inserting  an  itaa  a*  the  tacond 
alaaont  of  a  list  vaa  to  put  togothar  a  nov  list  consicting  of 
tha  first  alaMnt  in  tha  original  litt«  follovad  by  tna  nau 
alaaont  and  than  tho  raat  of  tha  original  Ifat.  Notica  that  «a 
can  dascriba  tha  algoritha  without  rafarring  to  function  naaaa. 

The  LISP  function,  1naort-*ocofid,  that  «a  uroto  aarliarj  la  and 
to  loplaaant  tha  algoritha. 

Additional  Llat  Hacil|Mi1at1on  rooetlona 

Hart  art  tnraa  na«  function*  utafui  for  aartipuUting  lift*. 

function  Call*  Value  raturnod  Op*ration 

(^p«id  b)  '(c))  (■  i  c)  put  tvo  or  aor*  ii*ts  togotnor 

into  •  iingl*  1i*t 

'<c  4  a))  (a  4  c)  r*v*r**  tr.  ord*r  of  the 

eleeenti  in  the  arguaent 

(laat  '(f  b  1))  (1)  rat^n  the  iiat  consuting  of 

tha  laat  itea  in  tha  arguaant 


Tnara  art  loaa  taportant  point*  to  be  aad*  about  aach  of 
that*  functions,  tna  function  a^paid  la  uaad  to  conatruet  tiata. 

It  takas  tvo  or  aor*  arguaanta,  aach  of  vhieh  auat  b*  a  Hat. 

Tha  funciic*  consiruCTs  a  nsv  Mat  containing  tna  aiaaonts  of 
each  of  tha  arguatnta.  Contidtr  the  following  axeeplta: 

(append  '(peoa)  '(carrott  eatery)  '(broccoli)) 

(peas  carrots  entry  broccoli) 

:>  (ippand  '(bovo  (Jack  Suo))  '((Aone  Tod)  Handy  tIcHord)) 

(Davi  (Jack  Suo)  (Ann#  Ted)  Manoy  diehard) 

Notica  that  tha  etaaentt  in  the  arguotnt  1itts  are  not  cnanpad. 
That  Is,  the  eteeents  Ooro,  (^ocN  Sue),  O^no  Tod),  Stondy,  Md 
HI  chard  acre  coooinod  into  •  ne«  list,  but  the  ttooa  Jeefc  end  Swo 
wore  not  taken  out  of  the  oooeddad  Hat  (Jack  Sua). 


Doe  to  cntbbck*  tbe  police  force  atst  isk  Its  MBbers 
to  nke  twice  &s  nsp  stops  st  virions  stores  ukUb 
thej  pitrol.  Eouever^  wbereis  they  used  to  stsrt  it 
tbe  closest  store  sod  flnlsb  st  tbe  sost  dlstaot  store» 
they  sow  Mist  start  it  tbe  Bost  dlstint  ind  visit  tbe 
stores  Id  tbe  reverse  order.  They  Bust  dc  this  twice 
wbereis  they  lued  to  Bike  o&ly  one  visit  per  shift. 

Write  1  foDCtloD  tbit  tikes  i  list  of  store  isBes  — 
e.g.,  <wlt  Bicys  iibertsois  7-11)  --  is  Input  snd  returis 
tbe  revised  order  of  pmtrol  stops. 


WWUWWWWWWUV  Liso  Intarprator  viadoH  ///////////////////// 


Torgot  (TEMT>:  revoro^ 


L1$P  motor 


Click  button  to  chongo  coatanto  of  laft  hand  windou. 
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search  routines  are  insensitive  to  font  changes  so  that  students  can  search  for  a 
phrase  that  vras  printed  in  boldface,  such  as  technical  terms  or  chapter  headings. 
Searches  art  also  insensitive  to  case  changes,  so  searches  will  not  fail  simply  because 
the  target  word  began  a  sentence.  Finally,  searches  are  also  insensitive  to  line  breaks 
and  page  breaks,  so  a  search  for  a  phrase  that  is  broken  across  a  line  or  a  page  will 
succeed. 

The  search  in  BATBook  differs  from  the  search  in  some  word  processing  programs 
in  that  the  displayed  material  in  BATBook  is  always  a  single  page.  Unlike  word 
processing  programs  which  display  a  “window”  on  the  file,  BATBook  always  displays 
pages.  These  page  breaks  are  constant  —  if  students  search  or  page  through  the 
text,  a  given  page  will  always  be  displayed  with  the  same  material  and  containing 
the  same  page  number.  In  this  way,  BATBook  pages  are  as  much  like  physical  book 
pages  as  possible,  with  the  added  advantages  for  the  student  that  they  can  be  easily 
paged  and  searched,  and  the  advantages  for  the  experimenter  that  it  is  possible  to 
know  at  any  point  in  the  problem  solving  what  page  the  subject  has  available  for 
reading  and  exactly  what  target  information  led  the  subject  to  that  particular  page. 

Another  way  that  students  can  search  previous  text  in  a  less  directed  fashion  is 
to  use  the  page  forward  and  backward  buttons  to  simply  flip  through  the  text  until 
a  useful  section  is  found.  In  some  experiments,  it  may  be  desired  to  restrict  paging 
of  the  text  while  working  on  problems.  BATBook  allows  the  experimenter  to  disable 
paging  while  the  student  is  working  on  a  problem  set.  In  this  way,  if  students  want 
to  find  something  in  the  text,  they  must  type  an  explicit  search  key  instead  of  just 
paging  forward  or  backward  until  they  find  a  useful  section. 

Students  find  the  ability  to  search  text  quite  useful  and  often  use  it  while  working 
on  exercises,  particularly  in  the  earlier  portion  of  each  chapter.  This  enables  them 
to  access  and  review  text  presenting  new  concepts  required  in  the  exercises  and  to 
find  examples  demonstrating  the  use  of  these  concppts.  Students  are  typically  quite 
effective  in  using  the  search  capabilities.  The  majority  of  searches  are  successful,  and 
students  typically  follow  unsuccessful  searches  with  a  successful  search  for  a  different 
key.  In  most  cases  students  appear  to  recognize  the  context  for  which  they  were 
searching.  A  typical  search  pattern  begins  with  a  “find  first”  followed  by  several 
“find  next”  clicks  in  very  quick  succession,  followed  by  the  student’s  resumption  of 
work  on  the  problem  using  the  information  contained  in  the  resulting  page  displayed 
in  the  Text  Window. 

The  assumption  in  the  design  of  BATBook  is  that  the  student  is  using  the  system 
to  learn  the  Jiew  material  contained  in  the  instructional  text  rather  than  as  a  tool  for 
browsing  reference  material.  For  this  reason,  search  of  the  text  is  restricted  to  the 
portion  of  the  text  the  student  has  read  so  far;  the  search  will  not  find  occurrences 
of  the  search  key  beyond  the  farthest  page  that  has  been  read. 
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It  is  important  to  note  that  any  of  the  content  of  the  instructional  text  may 
be  used  as  a  Key  to  search  the  text.  This  use  of  full  text  indexing  in  BATBook 
is  similar  to  the  search  facility  in  the  SuperBook  system  (Remde  et  al.,  1987). 
Remde  et  al.  argued  that  the  ability  to  use  any  content  of  the  text  as  a  keyword  for 
retrieval  is  an  important  feature  of  electronic  books.  A  difficulty  facing  information 
retrieval  is  that  users  often  describe  targets  in  different  terms  from  those  built  into  a 
system.  The  disparity  of  terms  naturally  chosen  by  users  of  systenu  creates  problems 
for  information  retrieval  systems  and  for  naming  commands  in  computer  systems 
(Furnas,  Landauer,  Gomez,  &  Dumais,  1984).  Therefore,  students  are  much  more 
likely  to  be  successful  in  retrieving  a  target  portion  of  text  if  they  can  use  whatever 
aspects  of  the  text  they  remember,  rather  than  relying  on  an  index  of  keywords  that 
the  author  hais  selected  as  important. 

The  approach  of  full  text  indexing  differs  from  Hypertext  systems  currently  re¬ 
ceiving  much  attention  (e.g.,  Conklin,  1987;  Halasz,  Moran,  &  Trigg,  1987;  Robert¬ 
son,  McCracken,  &  Newell,  1981).  Hypertext  requires  author-generated  links  be¬ 
tween  screens  of  information.  In  Hypertext,  certain  words  or  phrases  in  the  text  are 
marked,  indicating  that  a  related  section  of  text  is  available  and  can  be  accessed  if 
desired.  Our  goal  in  BATBook  is  to  provide  the  students  the  freedom  to  pursue  their 
own  paths  through  the  material.  We  do  not  want  to  draw  the  associations  between 
potentially  related  sections  through  hypertext  links.  Instead,  we  are  interested  in 
seeing  what  paths  the  students  themselves  choose  to  follow  and  which  sections  or 
problems  trigger  access  of  previous  sections.  Furthermore,  in  our  experiments  stu¬ 
dents  are  learning  to  solve  problems  in  a  new  topic.  An  important  component  of 
the  target  skill  is  knowing  what  information  is  relevant  in  each  problem  situation. 
Therefore,  we  want  the  responsibility  for  accessing  relevant  text  to  be  in  the  hands 
of  the  student. 

We  feel  that  BATBook  provides  an  interesting  extension  of  the  concept  of  an 
“electronic  book”  that  has  been  suggested  as  a  revolutionary  new  communication 
medium  (e.g.,  Weyer,  1985;  Yankelovich,  Meyrowitz,  &  van  Dam,  1985).  It  has 
been  argued  that  electronic  books  and  hypermedia  provide  readers  access  to  infor¬ 
mation  of  various  media  enabling  them  to  pursue  their  own  paths  through  “webs” 
of  connected  information.  BATBook  enables  students  to  access  information  they 
themselves  have  generated,  along  with  what  is  already  encoded  in  the  book.  Stu¬ 
dents  using  BATBook  can  retrieve  solutions  of  problems  along  with  the  text  when 
useful  in  solving  new  problems. 
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4  Solving  Problems  in  BATBook 

In  this  section,  we  describe  how  we  have  used  BATBook  to  teach  students  to  pro¬ 
gram  in  LISP.  This  version  relies  on  a  LISP  problem  solving  environment.  To  use 
BATBook  for  another  domain  would  require  adapting  this  problem  solving  envi¬ 
ronment.  For  example,  a  simple  interface  could  be  added  to  enable  students  to 
enter  equations  instead  of  LISP  expressions  to  work  on  algebra  or  physics  problems. 
Completed  solutions  to  word  problems  could  be  stored  away  and  searched  just  like 
the  LISP  problems  we  have  discussed.  Even  without  such  an  interface,  however,  it 
would  be  possible  to  use  the  current  version  of  BATBook  in  other  domains  by  having 
students  write  their  solutions  on  paper  and  use  the  system  to  read  and  search  the 
text  and  to  access  examples  contained  in  the  text. 

4.1  Constructing  a  Solution 

When  the  student  completes  the  reading  for  a  section,  BATBook  presents  the  as¬ 
sociated  set  of  exercises.  The  student  initiates  the  problem  sequence  by  selecting 
the  “Start  Problems”  button  in  the  Exercises  Window,  whereupon  the  first  problem 
in  the  set  is  displayed.  In  the  LISP  domain,  these  problems  require  writing  LISP 
programs.  Students  have  access  to  a  LISP  Interpreter  Window  which  contains  a 
specially  written  Common  LISP  interpreter.  This  interpreter  prevents  certain  t3rpe8 
of  drastic  errors  (for  example,  accidentally  redefining  built-in  LISP  primitives  or 
getting  trapped  in  infinite  loops)  and  contains  a  simplified  error  handler  that  avoids 
the  standard  LISP  “break  loop”  which  is  often  confusing  for  novices.  The  inter¬ 
preter,  like  the  other  facilities  in  BATBook,  also  records  every  student  keystroke 
and  mouse  click,  as  well  as  the  resulting  system  response. 

The  environment  also  contains  an  extremely  simplified  Emac8-b2ised  editor  which 
can  be  invoked  from  the  LISP  Interpreter  Window  to  edit  a  function  definition.  The 
editing  commands  are  invoked  from  labeled  function  keys  on  the  keyboard.  The 
editor  contains  commands  to  delete  a  character,  delete  a  line,  and  insert  a  deleted 
line.  The  arrow  keys  can  be  used  to  move  left  and  right  one  character  and  up  and 
down  one  line.  The  editor  also  contains  commands  to  save  the  function  and  enter 
the  revised  definition  into  LISP,  and  to  abort  the  edit  with  the  definition  unaltered. 
The  editor  prevents  the  student  from  saving  an  illegally  parenthesized  expression. 
Students  find  this  editor  relatively  easy  to  learn.  The  typical  scenario  for  solving 
problems  in  the  environment  involves  typing  a  function  definition  into  the  LISP 
Interpreter  Window,  trying  the  function  on  some  examples,  and  then  repeatedly 
editing  the  definition  and  trying  it  on  examples  until  it  works  properly. 
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4.2  Submitting  a  Completed  Problem 

When  each  problem  is  completed,  the  student  submits  the  solution  in  order  to  store 
it  fcr  future  use.  In  our  experiments,  the  submission  procedure  also  included  an 
analysis  of  the  solution  to  determine  whether  it  was  correct.  However,  this  is  an 
option  that  can  be  manipulated.  In  some  circumstances  it  may  be  desired  instead 
to  accept  all  student  solutions  without  checking  their  correctness. 

The  submission  process  is  initiated  by  clicking  the  “Submit”  button  that  appears 
beneath  the  problem  statement  in  the  Exercises  Window  (see  Figure  2).  BATBook 
then  prompts  the  student  to  enter  the  name  of  the  function  he  or  she  has  written 
to  solve  this  problem  (see  Figure  3).  BATBook  then  checks  the  solution  by  running 
the  student’s  program  on  several  test  cases  associated  with  each  problem.  If  the 
program  does  not  produce  the  correct  results  for  a  test  case,  BATBook  informs  the 
student  how  the  program  behaved  incorrectly  and  asks  the  student  to  try  to  fix 
the  solution.  The  student  is  required  to  attempt  to  fix  the  function  and  submit  a 
second  solution.  If  the  second  submitted  solution  is  still  incorrect,  the  student  is 
again  informed  but  this  time  is  given  the  option  of  continuing  with  the  next  problem 
or  continuing  to  try  to  fix  the  solution  (see  Figure  4). 

After  the  completed  solution  is  submitted,  a  new  problem  is  displayed  in  the 
Exercises  Window.  At  the  completion  of  the  exercises  within  the  problem  set,  the 
Exercises  Window  is  cleared  and  the  Text  Window  displays  the  page  that  begins 
the  next  section  of  text. 

4.3  Labeling  Completed  Solutions 

BATBook  contains  an  option  in  which  students  are  able  to  label  their  final  submitted 
solution  (whether  correct  or  incorrect)  for  the  problem.  In  this  version  of  the  system, 
when  students  submit  each  solution  they  are  asked  to  type  in  a  brief  label  to  describe 
it.  Students  are  told  that  the  label  can  be  used  at  a  later  time  to  retrieve  the  problem 
description  and  the  solution.  The  procedures  for  retrieving  solutions  are  described 
in  the  next  section. 

The  facility  to  label  a  completed  solution  enables  students  to  retrieve  their  past 
work  according  to  their  own  description  of  the  problem  and  solution  rather  than 
having  to  refer  specifically  to  the  content  of  the  problem  or  the  solution  itself. 
We  included  this  option  because  we  are  interested  in  seeing  whether  students  can 
profitably  use  the  labeling  facility  to  encode  their  solutions  selecting  features  useful 
for  later  retrieval. 
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Non  havt  ItarnM  to  Oofint  your  o»n  fiaictlont.  Tlw 
roitindar  of  this  chaptor  IntroOucM  mh  nt»  LICP  functlona  and 
givat  you  prictfca  putting  togithar  coablnatlona  of  functtona  In 
ordar  to  accoaplitn  aora  coaplicatad  tatu.  u  aa<  lantionae  in 
Cnaptar  t,  a  auostantial  part  of  looming  to  prograa  involvaa 
laarnlng  not  Just  ho>  fwictlona  aerO.  but  hoa  to  uoa  thaa  to 
coaputa  oasirad  rasults.  Tba  procadura  by  amch  a  function 
coaputaa  a  raault  la  callad  tna  atgontna  for  tna  function.  Por 
aaaapla,  our  algorltna  for  Inaartlng  an  itaa  at  tha  tocond 
alaoant  of  a  Hat  aaa  to  put  togathar  a  naa  11at  conaiating  of 
tna  first  alaaant  in  tna  original  Hat,  foHoaad  by  tba  naa 
alaaant  and  tnan  tna  ratt  of  tna  original  Hat.  notict  tnat  at 
can  dascrlba  tns  algorltna  althout  rafarrlng  to  function  ntoaa. 

Tha  Lisn  function,  tnaart-atoand,  that  aa  arota  ttrlltr,  it  atid 
to  laplaatnt  tna  algorltna. 

additional  Llat  Manipulation  Faaetlona 

Hart  art  tnrtt  naa  functions  utiful  for  tantpulating  Httt. 

Function  Cal  la  Valua  raturnad  Oparatlon 

(apptnd  '(a  k)  '(el)  <a  k  c)  put  tao  or  tort  Hats  togatnar 

Into  a  tingla  list 

*  *))  <*  1*  c)  rtvarta  tna  ordar  of  tna 

tlaatnts  In  tna  argiaant 

(last  '(g  hi))  (1)  rttirn  tna  Hat  consisting  of 


put  tao  or  tort  Hsta  togatnar 
Into  a  single  list 
rtvarta  tna  ordar  of  tna 
tlaatnts  In  tna  arg>atnt 
rttirn  tna  list  consisting  of 
tna  last  Itaa  in  tna  trguoant 


Tntra  art  sott  laportant  points  to  bt  soda  about  aacn  of 
tnssa  functlona.  Tha  function  appoM  It  usad  to  construct  lists. 
It  tanas  tao  or  aora  argutantt,  aach  of  ahlcn  aust  kt  a  list. 

Tna  function  constructs  s  naa  list  containing  tna  aitttntt  of 
aacn  of  tna  argutantt.  Consldar  tha  folloaing  anatplaa: 

=>  (apptnd  '(pats)  '(carrtta  calary)  '(hraccalD) 

(pass  carrots  calory  broccoli) 

=>  (apptnd  '(ktva  (Jack  tua))  '((Anna  Tad)  dandy  tldard)) 

(Oava  (Jacn  Sua)  (Anna  Tad)  Money  hi  chard) 

Notict  tnat  tna  tltttnts  in  tna  trguaant  lists  art  not  cnangtd. 
That  It,  tha  alottnts  kavt,  (Jack  Sua),  (Anna  Tad),  Mandy,  and 
Mehard  aara  cotoinad  into  a  naa  list,  but  tna  atoas  Jack  and  k 
uare  not  tatsn  out  of  tha  aabaddtd  list  (Jack  Sua). 


Doe  to  eatbkckk  the  police  force  anat  ask  ita  Meahera 
to  aake  tulce  aa  aanp  atopa  at  varioaa  atorea  uhich 
tlier  patrol.  loaaeuer,  Hhereaa  they  aaed  to  atart  at 
the  Closest  store  and  finish  at  the  aoat  distant  store, 
they  aoM  aoat  atart  at  the  aoat  diatant  and  via  it  the 
stores  la  the  reverse  order.  They  anst  da  this  talce 
Hliereas  they  nsed  to  aake  only  one  visit  per  shift, 

Hrlte  a  fnnctlon  that  takes  a  list  of  store  naaea  — 
e.f.,  (all  aacys  alhertsona  T-ll>  —  as  Inpnt  and  retnrns 
the  revised  order  of  patrol  stops. 


Pltass  typt  In  tha  naaa  of  tha  MAIN  function 
for  this  problat.  Than  prtss  ENTER. 

tain  function  naaa;  cutbac^ 

OSD 

rwn 

Click  button  ta  changa  cantanta  of  loft  hand  ulndou. 


Figure  3.  Submission  of  a  completed  solution. 
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Algontha* 

Non  you  hivt  Itirnid  to  Otflix  your  oun  fmetlont.  TM 
rtaamoor  of  thla  c^apta^  introoucaa  aoaa  naa  LISP  functlona  and 
givaa  you  practtca  putting  togathar  coab1nationa  of  functlana  In 
ordar  to  accoaplun  aora  coapUcataa  tatkt.  da  aai  santionao  in 
Otaptar  1,  a  auoatantial  part  of  laarning  to  prograa  Involvaa 
laarning  not  Juat  hoa  ftaictlona  aort.  Out  hoa  to  uaa  tnaa  to 
cooputa  oaalrad  raaulta.  Tha  procadura  by  anich  a  fvmctlon 
coaputaa  a  raault  ia  callad  tna  algentna  for  tna  function.  Par 
axaapla,  our  algorltha  for  Inaartlng  an  Itoo  aa  tha  aocond 
alooant  of  a  Hat  aaa  to  put  togathar  a  naa  Hat  conalatlng  of 
tha  flrat  alaaant  In  tha  original  Hat.  folloaod  by  tha  naa 
ataaant  and  than  tna  rant  of  tna  original  Hat.  Notica  tnat  ua 
can  daacriba  tha  algorltha  altnout  rafarring  to  function  naaaa. 
Tha  LISP  function,  Inoort-aacond,  that  aa  arota  aarllar,  la  aald 
to  laplaaant  tha  algorltha. 

Additional  Llat  Macilpulatloii  PaactlOAO 

Hart  ara  tnraa  naa  functlana  uaaful  for  aanipulating  llata. 


Function  Cal  la 


Valua  raturnad 


(VPMd  'ta  0)  "(O)  (a  a  e) 

'<c  d  a))  <0  at  c) 

Oaat'tgpD)  (O 


Oparatl on 

put  tao  or  aora  llata  togathar 
Into  a  amgla  Hat 
ravaraa  tna  ordar  of  tha 
alaaanta  in  tha  arguaant 
ratirn  the  Hat  conaiating  of 
tha  taat  Itaa  in  tha  arguaant 


Tnara  ara  aoaa  laportant  pointa  to  ba  aada  about  aach  of 
thaaa  functlana.  Tha  function  appand  la  uaad  to  eonatruct  Hata. 
It  tafcaa  tao  or  aora  arguaanta,  aach  of  ahich  ouat  ba  a  Hat. 

Tha  function  conatructa  a  naa  Hat  containing  tna  alaaanta  of 
aach  of  tha  arguaanta.  Conaldar  tha  folloalng  axaaplaa; 

:>  (appand  '(paaa)  '(orrata  calary)  '(braccall)l 
(paaa  carrota  calary  broccoli) 

:>  (appand  '(Pavn  (Jack  Sua))  '((Anna  Tad)  Handy  tKdvd)) 

(Oava  (Jack  Sua)  (Anna  Tad)  Handy  Diehard) 

Notlca  that  tha  aloaonts  in  tna  arguaant  Hats  ara  not  cnangad. 
That  It,  tha  alaaanti  bava,  (Jack  Sua),  (Anna  Tad),  Handy,  and 
Richard  uara  conblnad  into  a  nao  Hat,  but  tha  atoaa  Jack  and  S 
uaro  not  talon  out  of  tha  aaeocood  1iit  (Jack  Sua). 


Doe  to  cntbbcks  the  police  force  aut  lak  ita  aeaberd 
to  mke  twice  u  aiDy  dtopa  at  variona  atorea  wbicb 
tbey  patrol,  lowcver,  wbereaa  they  naed  to  atart  at 
tbe  cioseat  store  and  flnlab  at  tbe  Boat  diataat  atore, 
they  BOW  Boat  atart  at  tbe  Boat  diataat  aad  oiait  the 
stores  ia  tbe  reverse  order.  Tbey  Bast  do  this  twice 
whereas  tbey  ased  to  Bake  oaly  oae  visit  per  sblft. 

Krite  a  taactioa  that  takes  a  list  of  store  aaaea  — 
e.p..  (wii  Bacys  albertsoas  T-11)  —  aa  lapat  aad  retaras 
tbe  revised  order  of  patrol  stops. 


Tha  oaln  function  cutback  atlH  dooan't  nork.  I  haw  triad 
tha  arguaant  ('(NY.Casara  toyland  alT_nl|pitar)>  and  It  raturna 
the  valua  ((a11_n1^tar  toyland  Mf.Canara)  (all.nl^tar  toyland 
HT.Casarn)).  It  ahould  ratirn  (oll.nt^tar  toylaad  HV.Caaora 
an.n1tpitar  toyland  *r_CaBara>,  If  you  aould  Hka  to  contiiiM 
plaaaa  aodify  and  raaubait. 

(Plaaaa  grata  OK  if  you  aant  to  aodify  cutback.) 

(Plaaaa  praaa  SVMir  as  IS  button  if  you  can't  aolw  tnic 
problaa  and  aould  Hka  to  aow  on  to  tha  naut  probloo.) 


Subait  ts  It  I 


>  (dnfun  cutback  (ateraa)  (Mat  (ravaraa  atoraa))) 
catbaefc 

:>  (cutback  '(wlz  oacyb  blbbrtaanc  7-lS)> 

((7>11  bibbrtaona  bacya  w1z>> 

s>  (dbtun  cutback  (atorta)  (liat  (rtvaraa  atoraa)  (ravaraa  atoraa))) 
atback 

>  (cutback  '(ulz  aacya  alkartaona  7-11) > 

(S?-!!  albartaona  nacya  w1a>  (7-11  albartaona  nacya  wlx)) 


Click  kuttofi  t»  cliwif*  ecitcntc  cf  left  ktntf  y1ii#ow. 


Figure  4.  Feedback  on  a  submitted  incorrect  solution. 
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5  Retrieving  Past  Examples 

There  are  several  methods  for  retrieving  examples  in  BATBook.  Two  basic  types  of 
examples  can  be  retrieved:  examples  within  the  instructional  text  and  the  student’s 
own  previous  problem  solving  work.  The  examples  of  worked  solutions  embedded 
within  the  text  have  a  special  status  and  can  be  searched  using  a  method  different 
from  the  keyword  text  search  described  in  Section  3.2.  The  method  of  retrieving 
text  examples  is  described  in  Section  5.1. 

The  method  of  retrieving  the  student’s  own  work  depends  upon  the  options  set  by 
the  experimenter  or  instructor.  If  the  student  has  labeled  the  completed  solutions, 
these  labels  are  used  to  access  previous  solutions.  This  option  is  described  in  Section 
5.2.  If  the  students  were  not  asked  for  labels,  they  can  later  access  their  solutions 
by  using  a  probe  from  either  the  problem  description  or  the  content  of  the  solution 
itself.  This  method  is  described  in  Section  5.3. 

BATBook  provides  another  method  for  retrieving  previous  work  other  than  com¬ 
pleted  solutions.  Students  can  also  search  through  their  interactions  with  the  LISP 
interpreter.  This  type  of  search  is  described  in  Section  5.4. 


5.1  Searching  Study  Examples 

BATBook  contains  the  option  of  embedding  examples  within  the  instructional  text. 
We  refer  to  these  as  “study  examples”  to  distinguish  them  from  the  student’s  own 
solutions  to  problems.  A  series  of  study  examples  can  be  included  in  the  text 
similar  to  a  series  of  exercises  for  the  student  to  solve.  Each  study  example  includes 
a  problem  description,  a  solution,  and  an  explanation  of  the  solution  (see  Figure  5). 
The  examples  are  presented  on  the  right  side  of  the  screen  in  an  expanded  Exercises 
Window.  Rather  than  solving  the  problem,  the  student  simply  reads  the  example 
and  proceeds  to  the  next  example  by  selecting  the  “Next  Example”  button. 

The  student  can  search  through  any  of  the  previous  examples  at  any  time.  To 
initiate  a  search  of  examples,  the  student  first  selects  the  “Previous  Problems” 
button  beneath  the  LISP  Interpreter  Window.  This  changes  the  Text  Window  from 
Text  mode  to  Examples  mode.  The  examples  search  is  most  useful  while  solving 
problems,  so  it  is  important  that  the  Exercises  and  LISP  Interpreter  Window  remain 
in  view.  For  that  reason,  the  retrieved  examples  are  placed  in  the  Text  Window. 
The  student  then  types  a  search  key  opposite  the  “Target”  prompt  beneath  the 
window,  and  selects  the  “find  first”  or  “find  next”  button.  The  “find  first”  button 
finds  the  first  occurrence  of  the  search  key  in  any  of  the  examples  that  have  been 
read  so  far.  The  “find  next”  button  can  be  used  to  find  the  next  occurrence,  starting 
from  the  currently  displayed  match.  If  the  search  succeeds,  then  the  entire  example 
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tSSfllAi  HSf: _ Cfcaptcr  2:  LIST  Iwictioas 


Study  tht  follouirg  txMpIt  to  hov  ippond  It 
tflitingulthfd  froi  cofit  tnd  list.  Xt  It  Itportint  to  bt  tbit  to 
choott  tht  fight  ttf)ctlon  trot  ttong  th»tt  thrtt  In  constructing  t 
lift. 


=>  (list  *(•  b)  ^(e  t}) 

((t  D)  (c  d)} 

S>  (cofit  '{•  b)  'it  d)) 

((t  b)  c  d) 

=>  (tppind  '(a  b)  Me  tf» 

(a  b  c  d) 

Tht  function  cont  tiutyt  tiktt  t«o  argutortt  and  mttrtt  tht 
firtt  argutant  at  tht  b^inmng  of  tht  tocond.  rnt  function  Itat 
can  ttto  ont  or  tort  arguitnttj  and  aaktt  a  not  list  by  "trapping 
ptranthoata*  around  its  trguaontt.  Tht  trgjtent*  tty  bt  atott  or 
1ltt6«  and  Hat  prastrvts  tht  arguatntc;  that  is«  If  an  argutant 
If  a  list.  It  rtaains  an  tabtdded  lift  in  tne  new  list.  Tht 
function  ^iptnd  aates  a  not  11st  by 

frot  around  tacn  of  Its  trguatntt  and  putting  all  tht  tlttanta 
Into  ont  long  list.  (Actually,  Hat  and  apptnd  can  ot  callad  ultn 
no  argaaentt«  in  anich  casa  tach  touid  return  tht  etpty  Hat,  nil. 
In  addition  apptnd  can  bt  celled  with  1  argutant*  in  «hich  cast  it 
juat  rtturna  that  argutant.  Tht  occtflona  for  calling  thttt 
functions  In  this  tty  trt  rtro.  On  tht  othtr  hand*  Hat  can  be 
Quite  useful  ntn  one  argutant.  because  you  aigrit  aant  to  aaia  a 
list  containing  that  argutant.  ) 


Emtciscs 


2.7 

Doe  to  cutbacks  the  police  force  But  ask  Its  aeabers 
to  Bake  twice  u  aaor  (tops  at  varlou  stores  ublch 
thep  patrol.  loueoer,  ubereas  tbe^  ued  to  start  at 
the  closest  store  and  finish  it  the  nost  distant  store, 
they  now  nut  start  at  the  nost  distant  and  visit  the 
stores  la  the  reverse  order.  They  nut  do  this  twice 
whereu  they  ned  to  aake  only  one  visit  per  shift. 

Nrite  a  fuetton  that  takes  a  Hat  of  store  unea  — 
e.g.,  (wli  Bscys  albertsou  T-ll>  —  as  lapat  aad  retnru 
the  revised  order  of  patrol  stops. 

lere  Is  one  solntion  to  this  prohlcn; 
cXdefnn  patrol  (stores) 

(append  (reverse  stores)  (reverse  stores))) 


There  are  two  things  we  want  to  do  here:  we  want  to 
have  the  stores  visited  In  reverse  order  AHh  we  want 
It  done  twice.  He  know  of  the  fuctlon  "reverse''  wblek 
will  give  m  reversed  list.  By  calling  "reverse"  with 


rno  funetton  rovort*  totao  ooe  irguMiit,  uhleh  mat  bo  * 
11st.  It  rsturns  a  lift  in  vhlcn  tno  ordtr  of  tno  oltoonto  nos 
botn  rsvtrtto.  Novsvsr,  it  sots  not  rs.orss  tht  orOor  of  ony 
footOdod  tlsto. 

->  (rovwso  '(ontrt  *-c*s  oockt  shoos  jscSstl) 

(jockst  otioos  socks  drtss  Shirt) 

=>  (rsvsrss  'Isntrt  *-sss  (socks  saoss)  jsekst)) 

(Isckst  (socks  tnoss)  dross  shtrt) 

Ths  function  last  also  tokos  ont  orgusont,  sntch  oust  bo  a 
Hot.  Xt  rotumt  tno  toil  of  tnt  arguiont  contioting  of  tno  loot 
itso.  That  Is,  tt  dost  not  Just  raturn  tha  last  ttoa.  Inttoad,  It 
rotumt  0  list  sUh  one  oltsont  shich  It  ths  last  Ills  of  tha 
arguatni.  Calling  last  Is  squivolant  to  taking  surctsstut  c*'t  of 
a  not  until  only  ont  tlooont  roaaint. 


the  list  df  storu  tte  will  get  the  list  of  stores  Is 
reverse  order.  He  then  want  to  ue  the  fuctloa  "appead" 
to  put  two  reverted  eoplea  of  the  list  together  late 
one  long  Hat  of  atore  wnea. 

Then  me  cu  call  the  fuctloa  "patrol"  aa  foHout: 

n>  (patrol  *(wlz  ncys  albcrtaou  T-ID) 

(7-11  albertsou  Bacys  wli  7-11  albertaou  ucya  wit) 


torgot  (TEST):  rtoovlng  tbo  parontatoo^ 


CTicfc  button  to  ckonga  castonto  of  loft  bond  Mis 


Figure  5.  Reading  a  study  example  in  the  text. 
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(SSIMIlAi  LISP:  Chapter  2:  Defiittng  LISP  IttncLiMis 


2.8 

>  Mntt  to  kno»  thi  boot  ond  tha  oortt  otudont 

tn  taco  of  tho  diocutclon  ooctlono  of  bio  elooo.  Ho 
not  tno  noits  of  tno  participants  of  aacn  dltcuaalan 
group  In  lists  that  start  olth  tha  bast  otudont  and 
and  uith  tho  uorst.  brito  a  function  that  alii  taho 
sach  of  thasa  lists  —  a.g..  (salth  Jonss  saauals  blasa) 
—  and  raturn  anotnor  list  containing  only  tha  naaas 
of  tha  bast  and  tho  aorst  studant. 


Hsra  IS  ana  solution  to  this  praolaa: 

-Xdafun  bast-aorst  (studantsl 

(cons  (car  studants)  (last  studants))) 


This  function  takas  one  arguaant  ahlcn  It  a  list  of 
studant  naaas.  laa  can  uas  "car"  to  gat  tha  first  atudant'a 
nata  ond  “last*  to  got  a  11st  containing  tha  last  studant't 
nasa.  Noa  aa  cnooaa  tha  fidiction  "cons"  to  put  thasa 
tao  lists  togathar  bscausa  tha  first  studant'a  ntaa 
Is  an  atoa  and  can  slaply  ba  Inasrtad  into  tha  Hat 
containing  tha  last  studant't  nata.  This  alii  give 
us  our  tingle  list  of  tao  nattt. 

Than  aa  can  call  tna  function  "batt-aortt"  at  folloat: 

:>  (batt-aortt  '(ttith  ]onat  taautit  blaka)) 

(salth  blaka) 


3.1S 

1  pollster  has  collected  ansHere  to  a  serlea  of  qneatlaaa. 
These  ansuers  are  ia  the  Tara  of  a  list.  For  a  partlealar 
report  she  Hants  to  collect  ansHers  for  the  first  sad 
last  questions  given.  Nrite  a  fnnctlon  that  Hill  take 
a  list  of  ansHers  —  e.g.,  (31  pes  renting  deaocrat) 

—  and  retnra  the  first  and  the  last  ones  onl;. 


V\V\V\\\\VV\V\V\\\V\\  lisr  mtarpratar  HladOH  ///////////////////// 


(Mfun  pout  (antuart)  (car  tntwart)  (cant  last  tntuart 
alls 

>  (polls  '(21  pat  ranting  daaacrnt)) 

SPOS:  Unhaund  Vsriahia:  last 

:>  (safun  polls  (ontMtro)  (ear  anausrt)  (cons  (lost  sntHtrt))) 

alls 

>  (polls  '(21  pot  ranting  daoaerst)) 
kPOS:  Incarract  niabar  of  argt  to  cent 

>  (astun  polls  (snaHoro)  (Q 


I  tgtrnHiimfmaanMi 


Tar|«t  (PKOiLCMS):  pr«f««Mr 


Spipct:  [Tm  i 


ri  j  Jl'il  11  ■  JTI 11  -T  I^TFI  J,ll  lMd»| 


Cllek  kvttofi  %•  chpngp  ct«t«ntc  pf  Itft  htntf  wliitfoy. 


Figure  6.  Retrieval  of  a  study  example  from  the  text. 


16 


is  displayed  in  the  Text  Window.  If  the  search  key  cannot  be  found,  a  message 
is  displayed  and  the  student  can  either  edit  the  search  key  or  abandon  the  search. 
A  case  in  which  the  student  has  retrieved  a  study  example  while  solving  a  later 
problem  is  shown  in  Figure  6. 

Making  previous  examples  in  the  text  more  easily  available  to  students  is  an 
important  facility  of  the  BATBook  system.  Many  types  of  instructional  text  contain 
examples  to  demonstrate  a  concept  or  solution  method.  The  successful  “Minimal 
Manual”  approach  for  computer  manuals  developed  by  Carroll  and  his  colleagues 
relies  heavily  on  the  use  of  examples  to  demonstrate  the  sequence  of  actions  to 
achieve  a  particular  text  editing  goal  (Black,  Carroll,  &  McGuigan,  1987;  Carroll, 
Smith-Kerker,  Ford,  &  Mazur-Rimetz,  1987-1988).  Students  focus  on  examples  in 
textbooks  and  manuals  when  learning  new  procedures  (LeFevre  &  Dixon,  1986; 
Reder,  Charney,  &  Morgan,  1986;  Sweller  &  Cooper,  1985;  VanLehn,  1986;  Zhu 
&  Simon,  1987).  Furthermore,  the  type  of  elaborations  students  generate  when 
understanding  and  reviewing  examples  strongly  influences  their  success  on  later 
problems  (Chi  et  al.,  1989;  Pirolli  &  Bielaczyc,  1989).  Treating  the  example  as 
a  problem  to  be  solved,  in  which  each  step  can  be  “predicted”  and  the  reasons 
for  each  step  explained,  leads  to  better  learning.  Consistent  with  these  results, 
Charney  and  Reder  (1986)  demonstrated  that  students  learn  more  by  solving  a 
corresponding  problem  than  by  simply  foUowing  the  step  by  step  instructions  in  an 
example.  The  example  searching  capabilities  of  BATBook  facilitate  the  process  of 
learning  from  previous  examples  by  enabling  relevant  examples  to  be  more  easily 
retrieved  and  used  to  solve  new  problems.  As  an  experimental  tool,  the  example 
searching  capabilities  of  BATBook  enable  us  to  investigate  when  students  decide  to 
use  previous  examples  and  how  they  access  them. 

Another  potential  benefit  of  the  use  of  examples  in  BATBook  relies  on  the  way 
in  which  examples  are  distinguished  from  regular  expository  text.  This  may  serve 
to  focus  students  on  the  examples  and  encourage  them  to  process  the  examples 
more  appropriately  than  treating  them  as  additional  expository  text.  Scardamalia 
and  her  colleagues  have  argued  that  making  the  understanding  goals  more  explicit 
can  encourage  more  profitable  learning  strategies  (Scardamalia,  Bereiter,  McLean, 
Swallow,  ic  Woodruff,  1989). 

5.2  Searching  Previous  Solutions  Using  Labels 

Students  who  store  solutions  using  their  own  labels  can  later  retrieve  any  problem 
and  its  solution  using  the  label  as  a  key.  A  search  of  previous  solutions  can  occur 
at  any  point  the  student  chooses.  To  search  previous  solutions,  the  student  first 
selects  the  “Previous  Problems”  button  in  the  LISP  Interpreter  Window.  As  in  the 
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VtMJ*  L*KL :  kMt  mM  Morat  (tuAiit* 

*  profitsor  Mntt  to  kno«  tin  Mtt  iMI  ttio  writ  otudont 
lt>  Hcli  of  tho  ditcuiolon  toctlono  of  hit  c)nt.  Ho 
not  tho  noooo  of  tho  portlciponto  of  ooch  dlocuooloh 
gro>^  1n  lioto  thot  otort  mth  tho  boot  otudont  ond 
ono  otth  the  mrot.  brtto  o  fvnetton  that  atU  tabo 
each  of  thooo  lioto  —  o.g.,  (oatth  Jonoo  ooauoU  blako) 
—  and  roturn  anotnor  11ot  containing  only  tho  naaoo 
of  tho  boot  and  tha  oorot  otudont. 


Toui  saLunm: 

(defun  baat-«arat  (atidanta) 

(eana  (car  atuOonto)  (laat  atudanta))) 


k  patlBter  hu  collected  isaHeta  to  •  aetiea  of  qnestlou. 
These  aasuere  are  in  the  forn  of  a  Hat.  For  a  partlcalar 
report  ahe  Hants  to  collect  snsuers  for  the  first  and 
laat  questions  given.  Nrlte  a  fnnctloa  that  Hill  take 
n  list  of  nnsucra  --  a.g.,  (31  pea  renting  denocrat) 

—  and  return  the  first  and  the  last  anea  only. 


V\UU\\\\\\U\\\\\U  LISP  Intorprotor  HIndOH  ///////////////////// 


>  (oofun  pollator  (anoMora)  (Hat  (car  anauora)  (laat  anoHoro))) 
ellatar 

>  (pollator  '(21  yea  ranting  doaocrat)} 

(21  (doaocrat)) 

a>  (dafun  pollator  (anOHora)  (appaad  (car  anaaora)  (loot  aaaaoro))) 
ollotsr 

>  (pollator  '(21  yoa  ranting  doaocrat)) 

UM:  aoo*11ot  to  oppond:  21  nil 

:>  (dofun  pollator  (onOHoro)  ) 
llogol  dofun;  You  do  not  haeo  all  tao  nocooaary  caapanonta. 

>  (dafun  pollator  (onaHora)  (Q 


lAtfi  SfnncM. 


largat  (LAtCL):  atudant^ 


Click  button  to  ebango  contonto  of  loft  hand  am 
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search  of  study  examples,  this  replaces  the  Text  Window  with  a  Previous  Solutions 
Window.  The  student  can  then  type  part  or  all  of  a  previous  label  and  select  the 
“Label  Search”  button  to  initiate  the  search.  If  a  problem  with  a  matching  label 
is  found,  the  complete  problem  description  and  the  subject’s  final  solution  to  the 
problem  are  displayed  in  the  Previous  Solutions  Window  (see  Figure  7).  After 
finding  a  previous  solution,  the  student  can  return  to  the  LISP  Interpreter  Window 
to  attempt  to  map  a  component  of  that  solution  to  the  current  problem  or  decide 
to  search  for  a  different  problem  if  the  example  retrieved  turns  out  not  to  be  useful. 

Students  find  the  ability  to  retrieve  previous  solutions  a  very  useful  feature  of 
BATBook.  The  labeling  facility  provides  pedagogical  benefits  similar  to  the  facility 
for  retrieving  examples  from  the  text.  Much  of  problem  solving  early  in  a  stu¬ 
dent’s  learning  of  a  new  topic  relies  on  retrieving  memories  for  previous  solutions, 
adapting  the  procedure  to  fit  a  new  problem,  and  generalizing  the  result.  BATBook 
enables  students  to  encode  their  solutions  in  ways  that  later  can  be  used  to  retrieve 
them.  The  retrieval  process  is  greatly  facilitated  —  rather  than  having  to  recall 
a  solution  from  memory,  the  computer  enables  students  to  accurately  retrieve  the 
complete  problem  description  and  solution.  This  type  of  assistance  should  facilitate 
the  students’  abstraction  of  general  solution  plans  from  the  examples.  This  raises  a 
number  of  important  theoretical  issues,  which  we  are  currently  investigating  using 
BATBook:  What  type  of  similarity  causes  a  new  problem  to  remind  students  of  a 
previous  solution?  What  aspects  of  a  solution  do  students  encode  in  memory? 

There  are  potential  pedagogical  benefits  in  focusing  students  directly  on  the 
encoding  and  retrieval  question.  Students  know  when  they  label  a  solution  that 
their  labeling  will  affect  how  easy  it  will  be  to  retrieve  that  solution  at  a  later  time. 
Requiring  students  to  label  their  solutions  may  cause  them  to  reflect  on  the  solution 
and  try  to  abstract  what  was  important,  interesting,  or  novel  about  this  problem. 
This  type  of  reflection  is  generally  rnT'«!idered  an  important  part  of  sophistication 
in  a  domain  (e.g.,  Collins  &  Brown,  1988).  BATBook  provides  a  way  to  investigate 
these  issues  by  examining  students’  choices  of  labels,  whether  the  labels  can  be 
remembered  and  effectively  used,  whether  the  type  of  labels  change  with  expertise, 
etc. 


5.3  Searching  Previous  Solutions  Using  Problem  or  Solu¬ 
tion  Content 

A  different  method  for  searching  previous  solutions  is  available  to  those  students  not 
asked  to  label  their  solutions  when  submitting  them.  These  students  can  search  by 
referring  to  any  of  the  content  of  the  problem  or  the  solution.  In  this  method,  search 
is  initiating  by  selecting  the  “Previous  Problems”  button.  At  that  point,  the  student 
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Figure  8.  Retrieval  of  a  previous  solution  using  content  of  the  problem. 
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can  type  in  the  search  target,  and  select  the  “find  first”  or  “find  next”  button. 
BATBook  then  searches  through  the  problem  descriptions  and  their  solutions  for 
the  first  or  next  occurrence  of  the  search  string.  If  a  match  is  found,  the  problem 
description  and  the  subject’s  solution  are  displayed  in  the  Text  Window,  just  as  for 
a  Label  Search.  Only  the  problem  and  description  for  a  single  problem  are  displayed 
at  a  time.  An  example  of  this  t)rpe  of  search  is  shown  in  Figure  8. 

5.4  Searching  the  LISP  Interaction  History 

We  have  discussed  how  students  can  search  study  examples  embedded  in  the  text  or 
their  previous  solutions.  Another  potentially  important  type  of  information  is  the 
problem  solving  attempts  made  prior  to  a  solution.  Information  such  as  understand¬ 
ing  an  obstacle  encountered,  explanation  of  why  a  plan  failed,  or  other  unexpected 
events  in  pursuing  a  plan  may  play  an  important  role  in  providing  access  to  the 
event  in  memory  and  may  determine  what  can  be  learned  from  the  problem  solv¬ 
ing  episode  (e.g  ,  Carbonell,  1986;  Hammond,  1989;  Seifert  &  Hammond,  1989). 
For  this  reason,  BATBook  provides  students  the  ability  not  only  to  search  com¬ 
plete  examples  or  solutions  but  also  to  search  the  record  of  their  problem  solving 
interactions.  In  the  LISP  domain,  this  is  accomplished  by  a  facility  for  searching 
interactions  with  the  LISP  Interpreter. 

Students  can  search  the  history  of  their  interactions  with  the  interpreter  by 
selecting  the  “LISP  History”  button.  At  that  point,  the  student  can  type  a  search 
string  and  select  the  “find  first”  or  “find  next”  button.  As  in  search  of  the  text, 
this  search  finds  the  first  or  next  occurrence  of  the  search  string  in  the  log  of  the 
student’s  interaction  with  the  LISP  interpreter  for  that  chapter.  The  log  is  displayed 
in  the  Text  Window  with  the  target  string  in  reverse  video.  An  example  of  this  type 
of  search  is  shown  in  Figure  9. 

Students  find  the  search  of  the  LISP  history  to  be  a  useful  feature  while  solving 
problems.  Students  frequently  search  for  previous  episodes  in  which  they  encoun¬ 
tered  an  error  similar  to  one  appearing  in  the  current  problem  or  for  cases  when 
they  correctly  used  a  construct  relevant  for  a  current  problem. 
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6  Conclusions:  The  Study  of  Complex  Cognitive 
Skills 

In  this  paper,  we  have  described  the  BATBook  electronic  book  and  problem  solving 
environment.  BATBook  has  been  designed  as  both  a  pedagogical  and  an  experi¬ 
mental  tool.  The  pedagogical  benefits  of  the  system  derive  from  the  way  in  which 
BATBook  facilitates  students’  use  of  examples.  Study  examples  within  the  text  can 
be  easily  retrieved  and  used  when  the  student  faces  similar  problems  later  on,  or 
accessed  in  order  to  contrast  a  study  example  with  a  problem  in  which  a  somewhat 
different  solution  technique  is  required.  The  student’s  own  completed  solutions  can 
be  easily  retrieved  so  that  past  successes  can  be  applied  to  new  problems.  In  this 
way,  students  can  see  what  is  similar  between  problems  and  abstract  more  general 
strategies  and  plans.  Students  can  also  retrieve  a  previous  solution  to  contrast  it 
with  a  current  problem  to  consider  why  the  previous  technique  cannot  apply  to 
the  current  problem.  Furthermore,  focusing  students  on  the  encoding  and  retrieval 
process  itself  may  produce  pedagogical  benefits.  When  asked  to  label  a  solution, 
students  must  reflect  on  the  problem  solving  episode  and  extract  features  useful  for 
indexing  it.  In  general,  the  problem  solving  environment  encourages  students  to 
process  examples  differently  from  text,  and  to  consider  when  solving  a  problem  how 
it  relates  to  other  problems. 

As  an  experimental  tool,  BATBook  enables  us  to  examine  a  number  of  impor¬ 
tant  research  questions  facing  theories  of  learning  from  examples.  By  making  the 
students’  use  of  examples  overt  and  the  process  by  which  they  retrieve  examples 
explicit,  we  can  examine  in  greater  detail  the  role  that  examples  play  in  learning.  In 
particular,  we  are  investigating  the  reminding  and  use  of  examples.  What  t3q>e  of 
similarities  between  problems  cause  students  to  be  reminded  of  a  previous  solution? 
Are  students  distracted  by  superficial  similarities,  or  do  they  recognize  structural 
similarities  between  problems  with  different  surface  content?  At  what  point  dur¬ 
ing  problem  solving  do  remindings  occur?  What  type  of  information  is  encoded  in 
memory  from  a  problem  solving  episode? 

We  have  constructed  BATBook  to  examine  the  reminding  and  use  of  examples 
in  a  situation  in  which  students  are  engaged  in  learning  a  new  domain  and  solving 
problems.  Retrieving  a  study  example,  an  example  of  a  bug  encountered  previ¬ 
ously,  or  a  successful  solution  are  all  methods  in  the  student’s  battery  of  learning 
strategies.  In  this  way,  we  can  examine  the  \ise  of  examples  in  a  natural  context, 
where  the  examples  are  used  as  part  of  the  problem  solving.  This  approach  differs 
from  one  in  which  the  reminding  is  a  goal  explicitly  posed  by  the  experimenter, 
for  example  if  the  student  were  asked  to  describe  similarities  between  problems  or 
recall  information  that  was  only  encoded  for  the  purpose  of  recalling  it.  If  the  goal 
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is  to  characterize  how  people  remember  and  use  information  to  accomplish  goals,  it 
may  be  difficult  to  generalize  from  laboratory  tasks  in  which  the  subjects  are  not 
given  problem  solving  goals  (Seifert,  1988;  Seifert  &  Hammond,  1989).  In  order  to 
investigate  empirically  the  types  of  complex  learning  now  facing  cognitive  science 
theories,  it  will  be  necessary  to  move  beyond  simple  one- hour  laboratory  studies 
of  simplified  material  with  simple  “percent  correct”  performance  tests.  Instead, 
much  finer  grained  analyses  will  be  required  to  evaluate  learning  models.  Anderson 
(1987b)  has  argued  that  interactive  problem  solving  environments  that  record  and 
track  students’  reasoning  are  an  optimal  methodology  for  the  study  of  learning.  Ex¬ 
periments  such  as  our  BATBook  studies  (Paries,  1988;  Paries  &  Reiser,  1988;  Paries 
&  Reiser,  in  preparation)  and  instructional  experiments  based  on  intelligent  tutor¬ 
ing  systems  (e.g.,  Corbett  &  Anderson,  1989;  PiroUi,  1986;  Reiser,  Ranney,  Lovett, 
&  Kimberg,  1989;  Singley  &  Anderson,  1989)  provide  such  intensive  analyses  of 
students’  learning. 
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